2 research outputs found
Amortised Inference in Bayesian Neural Networks
Meta-learning is a framework in which machine learning models train over a
set of datasets in order to produce predictions on new datasets at test time.
Probabilistic meta-learning has received an abundance of attention from the
research community in recent years, but a problem shared by many existing
probabilistic meta-models is that they require a very large number of datasets
in order to produce high-quality predictions with well-calibrated uncertainty
estimates. In many applications, however, such quantities of data are simply
not available.
In this dissertation we present a significantly more data-efficient approach
to probabilistic meta-learning through per-datapoint amortisation of inference
in Bayesian neural networks, introducing the Amortised Pseudo-Observation
Variational Inference Bayesian Neural Network (APOVI-BNN). First, we show that
the approximate posteriors obtained under our amortised scheme are of similar
or better quality to those obtained through traditional variational inference,
despite the fact that the amortised inference is performed in a single forward
pass. We then discuss how the APOVI-BNN may be viewed as a new member of the
neural process family, motivating the use of neural process training objectives
for potentially better predictive performance on complex problems as a result.
Finally, we assess the predictive performance of the APOVI-BNN against other
probabilistic meta-models in both a one-dimensional regression problem and in a
significantly more complex image completion setting. In both cases, when the
amount of training data is limited, our model is the best in its class.Comment: This thesis served as the author's final project report for the
University of Cambridge part IIB Engineering Tripos. 37 pages, 7 figure
Amortised Inference in Neural Networks for Small-Scale Probabilistic Meta-Learning
The global inducing point variational approximation for BNNs is based on
using a set of inducing inputs to construct a series of conditional
distributions that accurately approximate the conditionals of the true
posterior distribution. Our key insight is that these inducing inputs can be
replaced by the actual data, such that the variational distribution consists of
a set of approximate likelihoods for each datapoint. This structure lends
itself to amortised inference, in which the parameters of each approximate
likelihood are obtained by passing each datapoint through a meta-model known as
the inference network. By training this inference network across related
datasets, we can meta-learn Bayesian inference over task-specific BNNs